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Abstract 

According  to  current  theories  of  cognitive  skill  acquisition,  new  problem  solving  rules  are  constructed  by 
procedural i/ation,  production  compounding,  chunking,  syntactic  generalization,  and  a  variety  of  other  mechanisms. 
All  these  mechanisms  are  assumed  to  run  rather  quickly,  so  a  rule's  acquisition  should  be  a  matter  of  a  few  seconds 
at  most.  Such  "learning  events”  might  be  visible  in  protocol  data.  This  paper  discusses  a  method  for  locating  the 
initial  use  of  a  rule  in  protocol  daa.  The  method  is  applied  to  protocols  of  subjects  teaming  three  »»*«•  a  river 
crossing  puzzle,  the  Tower  of  Hanoi,  and  a  topic  in  college  physics.  Rules  were  discovered  at  the  me  of  about  one 
every  half  hour.  Most  rules  required  several  teaming  events  before  they  were  used  consistently,  which  is  not 
consistent  with  the  one-trial  teaming  predicted  by  explanation-based  teaming  methods.  Some  observed  patterns  of 
teaming  events  were  consistent  with  a  learning  mechanbm  based  on  syntactic  generalization  of  rules.  Although 
most  rules  seem  to  have  been  acquired  at  impasses-occasions  when  the  subject  does  not  know  what  to  do  next- 
there  were  clear  cases  of  rules  being  teamed  without  visible  signs  of  an  impasse,  which  does  not  support  the  popular 
hypothesis  that  all  teaming  occurs  at  impasses. 

Introduction 

The  goal  of  this  research  is  to  see  if  people's  behavior  during  the  initial  construction  of  a  rule  sheds  any  light 
oo  which  of  the  many  contemporary  models  of  rale  acquisition  is  a  better  characterization  of  human  teaming 
methods.  Many  theories  of  cognitive  skill  acquisition  assume  that  rales  we  initially  formed  during  events  bating 
only  a  few  seconds.  This  is  about  the  same  time  scale  as  a  protocol  line,  so  such  "teaming  events"  might  be  visible 
in  a  protocol.  However,  it  is  difficult  to  locate  the  exact  line  of  a  protocol  where  a  rule  is  formed.  All  the  existing 
simulations  of  cognitive  skill  acquisition  (e.g.,  Anderson,  Farrell  A  Saurers,  1984;  Anzai  A  Simon,  1979;  Klahr, 
Langley  A  Neches,  1987;  VanLehn,  1983,  in  press)  use  multi-line  episodes  or  other  units  of  analysis  that  are  too 
large  for  this  purpose. 

This  paper  first  presents  a  method  for  locating  protocol  lines  where  rules  are  acquired,  then  discusses  the 
results  obtained  by  applying  it  to  protocols  from  three  task  domains.  Two  of  the  tasks  are  puzzles:  the  Tower  of 
Hanoi  and  a  river-crossing  puzzle.  Although  these  two  studies  were  intended  merely  to  check  that  the  analytic 
method  works,  they  nonetheless  yielded  some  interesting  findings.  In  the  third  study,  the  task  domain  is  college 
physics.  This  simulation  is  still  under  construction,  so  only  some  preliminary  results  can  be  reported.  All  three 
tasks  are  similar  in  that  the  subjects  are  "learning  by  doing"  -  they  are  solving  problems  without  help  from  a  tutor  or 
an  instruction  manual.  It  is  an  open  question  whether  the  results  and  methods  discuss  herein  extend  to  other 
instructional  situations. 

A  method  for  locating  protocol  lines  where  rules  are  learned 

The  following  two-step  method  is  used  to  locate  lines  of  the  protocol  corresponding  to  the  construction  of  a 
rate.  The  first  step  is  id  postulate  a  large  set  of  plausible  rales  for  problem  solving  in  the  task  domain.  The  rales  can 
be  inferred  from  analyzing  other  subjects’  protocob,  from  task  analysis,  from  interviewing  subjects,  from  one’s  own 
intuition,  from  writing  a  simulation  program,  or  from  any  other  f'Hirce.  Each  rule  is  written  with  the  weakest 
preconditions  possible  so  that  it  will  be  applicable  in  the  witkv  ix  ^ibte  range  of  situations.  The  second  step  is  to  fit 
this  vocabulary  of  rales  to  the  given  protocol.  At  almost  every  c  of  the  simulation,  there  will  be  many  rales  that 
can  be  ^plied  because  the  rules  have  weak  preconditions.  The  .  selects  one,  and  the  simulation  applies  it.  The 
user’s  job  is  to  find  a  sequence  of  rule  selections  that  maximizes  the  fit  of  the  simulation  to  the  protocol.  If  the 
protocol  can  be  formally  encoded,  this  step  can  be  automated  (VanLehn  A  Garlick,  1987;  Kowalski  &  VanLehn, 
1988).  The  result  is  a  table  which  aligns  the  protocol  with  rate  firings  and  "missed  opportunities'' -occasions  where 
a  rale  could  have  fired  but  did  not  For  instance,  in  Tables  1  and  4  below,  rows  correspond  to  protocol  lines  and 
columns  correspond  to  rales.  Cells  of  the  table  contain  a  "1"  if  the  rale  fired  during  that  line  of  the  protocol,  a  "0"  if 
it  could  have  fired  and  did  not,  and  a  blank  if  it  was  not  applicable  at  that  time. 

Given  such  a  table,  two  kinds  of  analysis  are  performed.  The  first  is  simply  to  look  for  patterns  in  the  firings 
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Table  1:  Protocol  and  simulation  of  a  subject  solving  a  river  crossing  problem 
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and  miiMpett  opportunities.  Using  this  kind  of  analysis,  it  was  discovered  that  rales  are  rarely  learned  completely  in 
one  trial.  Typically,  the  initial  firing  of  a  rule  is  followed  by  one  or  more  missed  opportunities,  then  another  firing,  a 

opportunity,  and  so  on  with  an  increasing  ratio  of  firings  to  missed  opportunities.  Such  patterns  have 
theoretical  implications.  For  instance,  gradual  acquisition  is  not  consistent  with  the  operation  of  learning  methods, 
such  as  explanation-based  learning  (e.g..  DeJong  A  Mooney,  1986),  that  acquire  rales  in  one  trial.  In  one  case,  the 
precise  pattern  of  gradually  increasing  frequency  of  usage  was  predicted  by  a  learning  mechanism  based  on 
syntactic  generalization  of  the  rule. 

Although  the  mere  pattern  of  rule  usage  has  shed  some  light  on  learning  mechanisms,  a  second  type  of 
analysis  has  proved  to  be  even  more  productive.  This  type  of  analysis  examines  the  subject’s  utterances  in  the 
vicinity  of  each  learning  event.  For  instance,  several  authors  have  claimed  that  all  learning  occurs  at  impasses 
(Laird,  Rosenbloom,  A  Newell,  1986;  VanLehn,  1988).  An  impasse  can  only  be  precisely  specified  relative  to  a 
given  simulation  model,  but  the  rough  idea  is  that  the  model  "doesn’t  know  what  to  do  next"  If  this  idea  is  taken  at 
face  value,  and  impasse-driven  learning  is  universal,  then  the  subject's  protocol  at  the  first  use  of  a  rule  should  show 
signs  of  confusion  or  hesitation  because  the  subject  is  at  an  impasse.  The  analyses  presented  below  show  that  most 
initial  uses  of  a  rule  are,  as  predicted,  accompanied  by  unusually  long  pauses  or  by  comments  such  as  "I  can't  do  it," 
or  "It's  not  that  easy."  However,  several  cases  were  found  where  there  are  no  such  signs  of  an  impasse  at  all.  Thus, 
the  data  are  consistent  with  the  claim  that  most  but  not  all  rule  acquisitions  are  triggered  by  impasses. 

Study  1:  A  river  crossing  puzzle 

The  protocol  analyzed  in  this  initial  study  was  not  collected  with  the  intention  of  studying  the  rule  acquisition 
process,  so  it  has  some  methodological  flaws.  However,  it  has  the  ex  positional  advantage  of  being  a  very  short 
protocol  that  nonetheless  demonstrates  some  of  the  paper’s  claims. 

The  subject,  a  9-year  old  girl,  was  given  standard  instructions  for  talk-aloud  protocols  then  asked  to  solve  the 
following  puzzle: 

Three  men  want  lo  crow  a  riv«r.  They  find  a  boat,  but  it  is  a  vary  mall  boat  It  will  only  K  id  200  pounds.  The  men 
are  named  Large,  Medium  and  Small.  Large  weights  200  pounds.  Medium  weights  120  pc  .aids,  and  Smell  weights  80 
pounds.  How  can  they  eh  get  wrote?  They  might  have  to  mike  several  trips  in  the  boat 

The  subject’s  protocol  and  an  analysis  of  it  appear  Table  1.  The  first  column  numbers  the  cycles  of  rale 
execution.  The  second  column  abbreviates  the  puzzle’s  state-the  notation  ”LMSbw"  means  that  Large,  Medium. 
Small  and  the  boat  are  on  the  source  bank,  and  nothing  is  on  the  destination  bank.  The  third  column  contains  the 
subject’s  protocol  The  remaining  columns  indicate  rale  firings  and  missed  opportunities.  The  major  rales  used  in 
the  simulation  wppeu  in  Table  2.  Rule  1  is  selected  for  firing  at  every  opportunity  (cycles  1,2, 5, 6  and  9).  The 
firing  of  the  rule  at  cycles  2  and  6  causes  a  subgoel  to  be  generated  because  the  boat  is  not  on  the  source  bank.  This 
assumes  a  cognitive  architecture  that  automatically  translates  precondition  violations  into  subgoals.  (The  simulation 
was  conducted  on  Teton  (VanLehn  A  Ball  in  press),  but  ACT*.  Sov,  OPS  and  many  other  architectures  have  this 
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Tabic  2:  Majors  rules  used  in  simulating  the  solution  of  the  river  crossing  puzzle 

1.  If  the  goal  is  to  move  some  men  from  bank  X  to  bank  Y,  and  the  boat  is  on  bank  X,  then  load  the  boat 
to  its  maximum  capacity,  sail  it  across,  and  unload  it. 


2.  If  the  goal  is  that  the  boat  be  on  bank  X,  and  the  boat  is  on  another  bank,  Y,  and  the  boat  requires 
someone  to  sail  it,  and  there  are  some  men  on  bank  Y,  then  load  the  boat  with  a  small  man,  sail  it 
across,  and  unload  it 


property.)  The  subgoal  of  getting  the  boat  back  to  the  source  bank  causes  the  application  of  rale  2  at  cycles  4  and  8. 

Although  all  applications  of  rale  1  go  smoothly,  the  first  application  of  rule  2  is  preceded  by  verbal  evidence 
of  an  impasse.  In  the  last  line  of  cycle  3,  the  subject  says,  "But  I  can’t  do  it  somebody  has  to  sail  the  boat 

back."  The  subject  says  she  is  stuck  while  at  the  same  time  mentioning  an  operation  that  could  be  applied.  My 
interpretation  of  this  line  goes  as  follows.  The  subject  recognizes  that  the  puzzle  situation  is  an  of 

reality,  but  she  is  unsure  about  how  much  of  an  idealization  it  is.  In  particular,  she  does  not  know  whether  the 
experimenter  intends  her  to  adopt  the  real-life  constraint  that  most  boars  require  a  helmsman.  The  puzzle 
instructions  do  not  state  this  constraint  (Indeed,  another  subject  choee  the  other  interpretation  of  the  inunctions, 
allowed  the  boat  to  sail  itself  back,  and  answered  that  it  takes  only  two  trips  to  get  an  the  men  across.)  On  this 
interpretation,  the  subject  already  "has"  rule  2,  but  she  does  not  know  whether  she  is  supposed  to  use  it  in  this 
puzzle.  After  the  experimenter  confirms  that  this  type  of  boat  requires  someone  to  sail  it,  the  subject  qiplies  the  rule 
(cycle  4).  Although  this  learning  event  is  arguably  not  rale  acquisition,  it  is  clearly  learning  of  some  kind.  As  will 
be  seat  shortly,  it  shares  several  properties  with  cases  that  are  quite  clearly  rule  acquisition  events. 

The  second  application  of  rale  2  is  also  preceded  by  signs  of  an  impasse  (cycle  7).  The  subject  again  claims 
to  be  stuck,  saying  "...can’t  think  of  anything...."  Apparently,  whatever  she  learned  during  cycle  3  is  not  immediately 
applicable  at  cycle  7.  As  will  be  seen  later,  this  is  quite  typical- the  first  firing  of  a  rale  is  followed  by  one  or  mote 
missed  opportunities.  Several  learning  mechanisms  are  consistent  with  this  behavior.  For  instance,  it  could  be  that 
the  rule  (or  assertion)  learned  at  cycle  3  has  such  a  highly  specific  precondition  that  it  does  not  apply  at  cycle  7,  so 
the  learning  mechanism  must  create  a  generalized  version  of  it  Another  possible  explanation  rests  on  context 
effects-when  the  new  rule  is  stored  in  memory,  it  is  indexed  in  pan  by  the  context  of  cycle  3,  which  is  assumed  to 
be  so  different  from  the  subsequent  retrieval  context  (cycle  7)  that  retrieval  fails.  Although  protocol  data  can 
differentiate  such  learning  mechanisms  (see  below  for  an  example),  this  particular  case  is  consistent  with  a  variety 
of  learning  mechanisms. 

Depending  on  which  learning  mechanism  one  believes  in,  cycle  8  represents  either  a  second  firing  of  rale  2 
or  the  firing  of  a  new  rule  that  is  a  generalization  of  rale  2.  As  mentioned  earlier,  this  research  method  uses  rales 
with  the  weakest  preconditions  possible,  so  in  the  simulation  of  this  protocol,  cycle  8  is  a  second  firing  of  rale  2. 
However,  this  is  just  a  methodological  device  for  locating  learning  events  in  protocols.  It  is  not  intended  as  a  claim 
about  detailed  learning  mechanisms.  I  will  continue  to  speak  as  if  general  rules  were  firing  intermittently,  even 
though  it  may  be  each  general  rule  is  actually  an  evolving  collection  of  specific  rales. 

In  summary,  this  protocol  shows  three  interesting  features.  (1)  The  initial  acquisition  of  the  rale  did  not 
suffice  to  make  it  reliably  operational.  A  second  learning  event  was  required  (cycle  7).  The  second  learning  event 
took  less  time  than  the  first  (cycle  3).  This  pattern  -  an  initial  formulation  of  a  rule  followed  by  one  or  more 
refinements  of  it  -  occurs  in  lata  protocols  as  well.  (2)  Both  learning  events  seem  to  be  triggered  by  an  impasse-a 
point  where  the  subject  does  not  know  what  to  do  next.  Impasse-driven  learning  has  been  touted  as  a  universal 
method  for  acquiring  rules  (VanLehn,  1988;  Laird,  Rosenbloom,  k  Newell,  1986).  It  will  be  seen  lata  that  although 
it  is  common  for  impasses  to  trigger  learning,  otha  types  of  triggering  can  also  occur.  (3)  The  subject  reported 
neitha  the  role  that  was  formed  nor  the  processes  that  constructed  it  The  existence  of  the  rale  can  be  inferred  from 
ha  action,  but  if  it  was  ever  present  in  ha  working  memory,  she  chose  not  to  mention  it  This  too  will  turn  out  to 
be  a  common  feature  of  learning  events. 
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Study  2:  The  Tower  of  Hanoi 

This  study  is  a  reanalysis  of  the  classic  protocol  of  Anzai  and  Simon  (1979)  wherein  the  subject  invents 
several  solution  strategies  for  the  five-disk  Tower  of  Hanoi  over  the  course  of  90  minutes.  During  this  time  she 
receives  no  instruction.  This  corrects  a  methodological  flaw  in  the  first  study,  where  the  experimenter’s  comments 
seem  to  have  been  instrumental  in  the  subject’s  learning. 

Anzai  aid  Simon  uncovered  the  major  strategies  that  the  subject  acquired  and  postu laird  learning 
mechanisms  sufficient  to  acquire  those  strategies.  They  did  not  attempt  a  line-by-line  comparison  of  the  protocol 
and  the  behavior  of  their  model.  Using  Teton  (VanLehn  St  Ball,  in  press),  we  found  that  additional  assumptions 
were  necessary  in  order  to  achieve  a  line-by-line  simulation  of  the  protocol.  The  moat  important  new  assumption  is 
that  the  subject  has  a  strategy  that  develops  around  the  first  ten  minutes  of  the  protocol  and  remains  fixed  throughout 
the  rest  of  the  protocol.  However,  this  strategy  gives  ambiguous  advice  on  25%  of  the  moves,  which  will  be  called 
the  major  moves.  On  the  major  moves,  the  subject’s  fixed  strategy  narrows  the  choices  down  to  two 
but  it  does  not  say  which  one  to  take.  The  bulk  of  the  subject’s  learning  consists  of  a  progression  of  strreegies  for 
making  these  decisions.  With  this  new  assumption,  a  model  was  formulated  that  fits  the  lines  of  the  protocol  with 
high  accuracy,  exceeding  even  that  found  in  Newell  and  Simon  (1972).  The  4— »»■  can  be  found  in  VanLehn  (1989). 

Table  3  shows  the  rules  used  to  make  the  mqjor  move  decisions.  Table  4  shows  the  analysis  of  the  protocol 
Each  row  of  Table  4  is  a  major  move.  The  first  column  numbers  the  nuyor  moves,  the  second  column  kbbrevirees 
the  puzzle’s  stare  just  prior  to  the  move,  and  the  third  column  abbreviates  what  tire  subject  said  while  w*«t™»g  the 
move.  The  puzzle's  pegs  are  labeled  A,  B  and  C,  where  A  is  the  peg  that  the  di*s  start  on  and  C  is  the  peg  they 
should  end  up  on.  The  disks  are  labeled  1  through  5,  with  5  being  the  largest  The  notation  *12534^."  meam 
that  disks  1 , 2  and  5  are  on  peg  A,  disks  3and4areonpegB,andpegCis  empty.  The  notation  *2B.  1A*  means  that 
the  subject  announced  a  goal  of  moving  disk  2  to  peg  B,  then  announced  a  movement  of  disk  1  to  peg  A.  The 
notation  "4pC  indicares  a  goal  of  moving  a  pyramid  or  group  of  four  disks.  Sometimes  the  subject  announces  a 
series  of  goals,  pauses,  and  announces  a  different  series  of  goals.  This  behavior  is  indicated  by  placing  two  rows  in 
the  table,  one  for  each  series  of  goals,  and  placing  ditto  marks  in  the  first  two  cells  of  the  second  row.  Horizontal 
lines  in  (he  table  indicate  places  where  the  subject  reset  the  puzzle  to  an  initial  state.  The  rightmost  six  columns  of 
Table  4  show  the  applicability  of  rules.  As  always,  a "  1"  indicates  a  rule  firing,  a  "0"  indicates  a  missed  opportunity, 
and  a  blank  indicates  that  the  rule  was  not  applicable.  A  T  indicates  that  the  rule  may  or  may  not  have  been  fired  - 
the  protocol  evidence  is  unclear.  The  asterisks  will  be  explained  in  a  moment. 


Table  3:  Abbreviations  and  descriptions  of  rules  for  handling  major  moves 


Initial  rules 

•  Look  The  Anzai  and  Simon  look-ahead  search  strategy. 

•  1  blk  If  the  goal  is  to  move  a  disk  from  one  peg  to  another,  and  there  is  a  single  disk  blocking  the  move, 
then  get  the  blocking  disk  to  the  peg  that  is  not  involved  in  the  move. 

•  2  blk  If  the  goal  is  to  move  a  disk  from  one  peg  to  another,  and  the  2-high  pyramid  (Le.,  disks  1  and  2) 
blocks  the  move,  then  get  disk  1  to  one  of  the  pegs  involved  in  the  move  (thus  freeing  disk  2  to  move  to 
the  peg  not  involved  in  the  move). 

Rules  acquired  during  the  protocol 

•  4B  Before  attempting  any  of  the  top  level  goals,  try  to  get  disk  4  to  peg  B. 

•  Dsk  (The  Anzai  and  Simon  disk  subgoaling  strategy.)  If  the  goal  is  to  get  a  disk  from  one  peg  to 
another,  and  there  are  some  disks  blocking  the  move,  then  get  the  largest  blocking  disk  to  the  peg  that  is 
not  involved  in  the  move. 

•  Pyr  (The  Anzai  and  Simon  pyramid  subgoaling  strategy.)  If  the  goal  is  to  move  a  pyramid  from  a  peg  to 
another  peg,  then  get  the  next  smallest  pyramid  to  the  peg  that  is  not  involved  in  the  move. 
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Table  4:  Rule  firings  and  missed  opportunities  for  the  major  moves 


Stele 

Preteeei 

Inttlel  rulee 

Leerned 

tHlSS _ 1 

rem 

inn 

an  li 

KUEnj 

icra 

Bmami 

1 

0  0 

0 

BISfIS  1 

1 

t 

7 

0  0 

0 

1 

•  1  •  0 

_ 2 _ 

ic 

1 

0  0 

0 

5 

45.  12.  3 

48.  1A 

0 

1 

•  1  •  0 

0 

« 

S.  4.  123 

18 

1 

0 

0 

7 

125. 34.  _ 

1C 

0 

1 

0 

0 

S 

,  1234.  5 

1 A 

1 

0 

0 

9 

3.  4.  125 

18 

0 

t 

0 

0 

1  0 

123.  _.  45 

3C,  1C 

0 

1 

0 

0 

1  1 

12.  .  345 

IB 

0 

1 

0 

0 

Dl 

1.  , 

wmmsMmassmi 

D1 

12.  . 

18 

in 

123.  _.  _ 

X.  28.  1C 

0 

0 

•  i  • 

0 

iff 

.  12.  3 _ 

Quit* _ 

? 

1 

o  _ 

5 

■a 

m  Jtuiaii 

0 

0 

1 

0 

lu 

0 

1 

0 

0 

iff 

1C 

0 

1 

0 

0 

H 

yWM 

0 

0 

•  1  • 

0 

In 

9 

1 

0 

0 

20 

12345.  . 

X,  48.  X.  28.  1C 

0 

0 

0  1 

0 

21 

45.  12.  3 

48.  1A 

0 

1 

1  0 

0 

• 

* 

48.  X.  1A 

0 

0 

1  •  1  • 

0 

5.  4.  123 

X.  38.  2A.  1A 

0 

1 

1 

0 

125. 34.  _ 

1C 

0 

1 

0 

0 

1234.  5 

4C.  3A.  28.  1A 

0 

0 

1 

0 

23 

3.  4.  125 

IB 

0 

1 

0 

0 

26 

123.  _.  45 

X.  28.  1C 

0 

0 

1 

0 

eh 

12.  .  345 

18 

0 

1 

0 

0 

28 

12345.  . 

X.  48.  X.  28.  1C 

0 

0 

0  1 

0 

29 

45.  12.  3 

48.  1A 

0 

1 

1  0 

0 

30 

5.  4.  123 

3pB.  18 

0 

0 

0 

1 

31 

125. 34. _ 

1C 

0 

1 

0 

0 

32 

1234.  5 

4pC.4C.3pA.3A,2C.1A 

0 

0 

1 

1 

33 

3.  4,  125 

2pA.  18 

0 

0 

0 

1 

34 

123.  _.  45 

3pC.  2p8,  1C 

0 

0 

0 

1 

KOI 

2oC.  18 

0 

0 

0 

\ 

Some  interesting  findings  are  visible  in  the  patterns  of  firings  and  missed  opportunities.  As  in  the  river 
crossing  puzzle,  it  is  never  the  case  that  a  rule  is  used  consistently  after  it  is  first  acquired.  Instead,  the  usage  of  a 
new  rule  increases  gradually.  A  second  observation  is  that  this  subject  occasionally  compares  the  results  of  an  old 
rule  with  those  of  the  rule  that  supplants  it  This  can  be  seen  in  both  cases  where  the  subject  re-does  the  planning  of 
a  move  (moves  18  and  21).  It  can  also  be  seen  in  move  32  where  the  subject  mixes  pyramid  goals  with  disk  goals. 
Thus,  we  do  not  see  a  rapid  transition  from  an  old  rule  to  a  new  one,  but  a  gradual  transition  that  is  sometimes 
accompanied  by  deliberate  comparison  of  the  old  and  new  rules. 

Obtaining  further  insight  into  the  character  of  the  rule  acquisition  process  requires  examining  the  protocol  in 
the  vicinity  of  the  initial  occurrences  of  the  rules  (see  VanLehn,  1989,  or  Anzai  &  Simon,  1979,  for  the  protocol 
itself).  In  the  river  crossing  study,  there  were  signs  of  impasses  at  both  the  initial  firing  of  rule  2  and  the  subsequent 
firing.  In  this  study,  impasses  were  also  present  at  most  of  the  early  rule  firings.  Asterisks  are  used  in  Table  4  to 
mark  rule  firings  that  were  accompanied  by  long  pauses  and  negative  comments,  such  as  "It’s  not  that  easy”  or  "I 
should  have  moved  3  to  C.”  Impasses  were  common  in  the  acquisition  of  both  rule  4B  and  the  disk  subgoaling  rule. 
However,  there  seem  to  have  been  no  impasses  involved  in  the  learning  of  the  pyramid  subgoaling  rule.  At  the  first 
firing  of  the  rule  (move  30),  the  subject  simply  started  phrasing  her  goals  in  toms  of  pyramid  instead  of  disks. 
Instead  of  saying  "3  will  have  to  go  to  B..."  as  she  said  at  move  22,  she  said,  "I  only  need  move  three  blocking  disks 
10...B."  There  seems  to  have  been  no  impasse  here.  At  32,  the  subject  said 

I  will  move  tha  lamenting  four  from  B  to  C...  It's  just  like  moving  four,  isn't  it?  So...  I  will  have  to  move  4  from  B 
to  C...  For  that,  the  three  that  we  on  top  have  to  go  from  B  to  A...  Oh,  yeah.  3  goes  from  B  to  Al  For  that,  2  has  to  go 
from  B  to  C,  far  that,  1  has  id  go  from  B  to  A. 
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Although  this  segment  is  long,  it  contains  none  of  the  signs  of  consternation  that  mark  the  other  learning  events. 

the  subject  seems  to  have  been  excitedly  comparing  the  disk  rule  and  the  pyramid  rule  and  proving  to 
herself  that  they  generate  the  same  plan.  If  an  impasse  is  defined  to  be  an  occasion  when  the  subject  does  not  know 
what  to  do,  then  this  segment  is  not  an  impasse,  for  the  subject  seems  to  have  two  alternatives  and  believe  that  both 
are  equally  correct.  In  short,  it  seems  that  most  rule  acquisitions  (2  of  3)  are  triggered  by  impasses,  but  rules  can 
sometimes  be  learned  without  impasses. 

There  is  a  subtle  pattern  in  the  acquisition  of  the  disk  subgoaling  rule.  Some  of  the  early  firings  of  the  rule 
are  marked  by  pauses  and  other  signs  of  impasses,  and  some  are  not  Although  space  does  not  permit  a  detailed 
minarinn  0f  the  data  (see  VanLehn,  1989),  it  appears  to  be  the  case  that  the  subject's  initial  formulation  of  the 
disk  rule  is  highly  specific  in  that  it  mentions  the  particular  disks  and  pegs  involved  in  the  major  move  where  it  is 
acquired.  Subsequent  applications  of  the  rule  cause  the  names  of  specific  pegs  and  disks  to  be  replaced  by  variables. 
This  gradual  generalization  of  the  rule  means  that  some  major  moves  can  be  handled  by  the  evolving  rule,  while 
others  cannot  and  force  the  rule  to  be  further  generalized.  Pauses  and  other  signs  of  impasses  correlate  perfectly 
with  the  places  where  generalization  is  predicted  to  occur.  In  particular,  if  it  is  assumed  that  the  subject  follows  the 
policy  of  generalizing  just  enough  to  get  the  rale  to  accommodate  the  present  situation,  then  it  will  take  four  learning 
events  to  lean  a  folly  general  version  of  the  disk  subgoaling  rale.  All  four  of  these  predicted  learning  events  are 
marked  by  impasses  in  the  protocol  (two  occur  during  move  14,  and  both  are  marked  by  a  distinct  pause).  So  it 
appears  that  impasse-driven  syntactic  generalization,  which  has  played  an  important  role  in  several  models  of  skill 
arqiridtinn  (e.g.,  VanLehn,  1983, 1986,  in  press),  seems  to  be  behind  the  acquisition  of  the  disk  subgoaling  strategy. 

A  last  point  to  mention  is  that  rales  were  discovered  at  a  rate  of  about  one  every  half  hour  (three  were 
discovered  in  the  90  minute  protocol).  This  rate  seems  to  hold  in  the  next  study  as  well 

Study  3:  College  Physics 

The  protocols  for  the  third  study  come  from  a  study  by  Chi,  Bassok,  Lewis,  Reimann  and  Glaser  (in  press)  of 
eight  students  learning  college  physics  from  a  standard  textbook.  Chi  et  al’s  study  used  a  training  format  that  comes 
close  to  the  way  students  learn  physics  in  college,  except  that  the  subjects  could  only  refer  to  a  textbook;  they  could 
not  ask  questions  of  a  teacher.  The  subjects  first  learned  the  initial  four  chapters  of  a  standard  college  physics 
textbook  to  criterion.  They  then  read  the  fifth  chapter,  which  covers  the  target  subject  matter.  Newtonian  particle 
dynamics.  When  they  came  to  the  worked  examples  at  the  end  of  the  chapter,  protocol  collection  began.  Protocols 
were  collected  as  the  students  studied  3  examples  ar-:  worked  19  problems.  The  examples  and  the  problems  present 
ample  opportunities  for  learning  because  they  addrt  issues  that  simply  are  not  covered  anywhere  in  the  preceding 
material.  For  instance,  the  concept  of  a  "normal  for. . '  is  first  introduced  in  the  context  of  an  example.  The  students 
took  between  8  and  29  hours  to  complete  the  study.  The  protocols  cover  the  last  3  to  6  hours. 

As  simulations  are  currently  being  constructed  for  each  of  the  8  protocols,  it  is  too  early  to  report  accurate 
data  on  learning  events.  However,  Bernadette  Kowalski  and  I  could  not  resist  doing  a  hand  analysis  of  one  protocol. 
We  found  clear  indications  of  five  rules  being  acquired.  As  the  protocol  lasts  3.3  hours,  this  is  an  average  of  one 
rale  every  40  minutes,  which  is  comparable  to  the  rate  found  in  the  Tower  of  Hanoi  study  (one  rale  per  30  minutes). 
We  found  some  evidence  that  rales  are  acquired  gradually,  but  we  are  reluctant  to  put  a  number  on  it  because  it  is 
difficult  to  detea  missed  opportunities  without  a  simulation.  The  usual  signs  of  impasses  marked  the  initial  firing  of 
3  of  the  3  rales.  The  other  two  rales  seem  to  be  acquired  as  the  subject  reflects  on  a  just-completed  solution.  As  an 
illustration,  the  next  few  paragraphs  present  one  of  these  rale’s  acquisition  event 

Subject  101  is  confused  about  the  difference  between  weight  and  mass  throughout  most  of  the  experiment 
(Many  other  students  had  the  same  confusion.)  Eventually  he  discovers  that  weight  is  the  force  due  to  gravity  while 
he  is  solving  the  following  problem:  "A  fireman  weighing  160  pounds  slides  down  a  vertical  pole  with  an  average 
acceleration  of  10  feet  per  second.  What  is  the  avenge  vertical  force  he  exerts  on  the  pole?"  The  subject  reads  the 
problem,  then  says: 

€.  Okay.  On,  wa'd  hava  to  considas. . . tha  forca  of  gravity. 

7.  Okay.  hat's  find  out  what  tha  forca  of  gravity  ia  axarting 

8.  on  th—  and  than  wa  can  figura  out  what,  what  his,  what  ha's 

9.  assarting  on  it. 

10.  Mow,  lat  aa  ramaabar,  waight  is  aqual  to,  what's  forca  aqual 

11.  to,  waight? 

12.  rorca  ia  aqual  to  waight  ovar  gravity  tiaaa  accalaration . 
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The  subject  decides  to  follow  a  generic  plan,  which  he  has  used  many  times  before.  The  plan  is  to  find  the  forces 
acting  on  the  body  (the  fireman,  in  this  case),  sum  them,  and  apply  F*ma.  He  summarizes  his  intentions  in  lines  7. 8 
and  9.  However,  the  plan’s  fust  goal,  which  is  to  find  the  force  of  gravity  acting  on  the  fireman,  thwarts  him.  He 
does  not  know  that  the  160  pound  weight  is  the  force  of  gravity  on  the  fireman,  so  he  sets  about  to  calculate  the 
force  using  the  derived  law,  F=<W/g)a.  After  fumbling  with  the  units  and  looking  up  the  appropriate  value  for  the 
gravitational  acceleration,  g,  he  substitutes  the  freefall  acceleration  for  a  and  obtains  F*(W/g)g»W.  At  this  point,  he 
says: 

42.  Oh,  Z'm  going  to  got  fore*  la  ogual  to  weight  divided  by 

43.  gravity  tlaaa  gravity  which  ia  going  to  be  equal  to  weight. 

44 .  Right? 

45.  Za  that  right? 

44.  Okay.  Oe,  ao  I'm  going  to  gat  140  pounda.  That' a  tha  forca. 

47.  Yaah.  Zt  kind  of  sakaa  aanaa  'cauaa  thay,  thay  walght  you  in 

48.  pounda,  don't  thay? 

49.  That' a  forca. 

50.  Okay.  3o,  avaraga  accalaratioa,  tha  forca  ha,  tha  gravity  ia 

51.  enacting  on  him,  yaah,  yaah,  that  makes  aanaa,  is  140. 

At  line  43,  he  has  the  solution  to  his  subgoal.  He  double  checks  the  math  in  lines  44  and  45  (probably),  and  again 
states  the  solution  in  line  46.  Although  he  could  simply  go  on  to  the  next  step  in  his  plan,  the  simplicity  of  the 
equation  F«W  apparently  prompts  him  to  reflect  on  his  solution.  Thus,  s  learning  event  begins  around  line  47.  The 
subject  appears  to  use  a  kind  of  explanation-based  reasoning.  Although  he  has  just  built  a  proof  that  F»W  far  this 
problem,  he  adds  a  second  "prooF  based  on  the  units  of  force  and  weight  (both  are  measured  in  pounds).  This 
seems  to  be  critical  to  establishing  the  generality  of  the  result,  which  is  that  weight  is  the  force  exerted  by  gravity  on 
an  object.  The  learning  event  ends  at  line  49,  and  the  subject  returns  to  the  plan  in  line  50.  However,  he  indulges  in 
one  last  check  of  the  result,  in  line  51,  before  going  on  to  finish  the  problem  off.  The  next  time  he  has  an  opportunity 
to  apply  his  new  rule,  he  initially  fails  to  retrieve  it,  but  is  reminded  of  it  halfway  through  the  problem,  and  happily 
applies  it.  Thereafter,  he  always  uses  the  new  rule  whenever  it  is  applicable. 

This  segment  of  the  protocol  illustrates  that  rule  acquisition  in  a  knowledge-rich  context  has  much  the  same 
character  as  it  does  in  the  knowledge-lean  context  of  learning  to  solve  a  puzzle.  For  instance,  it  appears  that  the 
acquired  rule  is  not  completely  learned  during  the  first  instance  of  its  use,  for  the  subject  nearly  misses  the 
opportunity  of  applying  it  later.  This  particular  rule  does  not  seem  to  be  acquired  at  an  impasse.  Instead,  the  subject 
seems  to  infer  it  while  reflecting  on  his  just-completed  solution  to  a  subproblem  (line  47).  However,  other  physics 
rules  do  seem  to  be  learned  at  impasses. 

Conclusions 

Three  analyses  have  been  presented  showing  that  the  initial  uses  of  problem  solving  rules  can  be  located  in 
protocol  data.  This  analysis  method  yielded  the  following  observations  about  the  acquisition  of  rules: 

1.  Rules  are  seldom  completely  learned  in  one  trial.  The  initial  firing  of  a  rule  is  often  followed  by 
several  missed  opportunities  before  the  rule  comes  to  be  fired  at  every  opportunity. 

2.  Sometimes,  this  gradual  increase  in  applicability  is  consistent  with  a  learning  mechanism  (VanLehn, 

1983,  in  press)  that  operates  by  initially  constructing  a  highly  specific  rule  then  generalizing  it  only 
when  an  impasse  forces  it  to. 

3.  Long  pauses,  negative  comments  and  other  signs  of  impasses  are  common  at  the  early  firings  of  rules, 
but  some  rules  are  acquired  without  any  visible  signs  of  an  impasse. 

4.  Sometimes,  the  subject  explicitly  compares  a  new  rule  to  the  old  rale  that  it  replaces.  This  indicates 
that  the  subject  is  probably  aware  of  both  of  them,  although  none  of  the  subjects  in  any  of  these  studies 
explicitly  mentions  or  describes  their  rules.  It  also  indicates  a  more-or-less  deliberate  application  of  a 
method  for  improving  one’s  knowledge. 

5.  In  the  context  of  learning-by-doing,  wherein  the  subject  receives  no  instruction  from  tutors  or  manuals, 
rule  acquisition  occurs  at  the  rate  of  about  one  rule  every  half  hour. 
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